<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
<title>NUMA</title>
<link rel="stylesheet" href="../../../../../doc/src/boostbook.css" type="text/css">
<meta name="generator" content="DocBook XSL Stylesheets V1.75.2">
<link rel="home" href="../index.html" title="Chapter&#160;1.&#160;Fiber">
<link rel="up" href="../index.html" title="Chapter&#160;1.&#160;Fiber">
<link rel="prev" href="speculation.html" title="Specualtive execution">
<link rel="next" href="gpu_computing.html" title="GPU computing">
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
<table cellpadding="2" width="100%"><tr>
<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../boost.png"></td>
<td align="center"><a href="../../../../../index.html">Home</a></td>
<td align="center"><a href="../../../../../libs/libraries.htm">Libraries</a></td>
<td align="center"><a href="http://www.boost.org/users/people.html">People</a></td>
<td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td>
<td align="center"><a href="../../../../../more/index.htm">More</a></td>
</tr></table>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="speculation.html"><img src="../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../index.html"><img src="../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="gpu_computing.html"><img src="../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
<div class="section">
<div class="titlepage"><div><div><h2 class="title" style="clear: both">
<a name="fiber.numa"></a><a name="numa"></a><a class="link" href="numa.html" title="NUMA">NUMA</a>
</h2></div></div></div>
<p>
      Modern micro-processors contain integrated memory controllers that are connected
      via channels to the memory. Accessing the memory can be organized in two kinds:<br>
      Uniform Memory Access (UMA) and Non-Uniform Memory Access (NUMA).
    </p>
<p>
      In contrast to UMA, that provides a centralized pool of memory (and thus does
      not scale after a certain number of processors), a NUMA architecture divides
      the memory into local and remote memory relative to the micro-processor.<br>
      Local memory is directly attached to the processor's integrated memory controller.
      Memory connected to the memory controller of another micro-processor (multi-socket
      systems) is considered as remote memory. If a memory controller access remote
      memory it has to traverse the interconnect<sup>[<a name="fiber.numa.f0" href="#ftn.fiber.numa.f0" class="footnote">8</a>]</sup> and connect to the remote memory controller.<br> Thus accessing
      remote memory adds additional latency overhead to local memory access. Because
      of the different memory locations, a NUMA-system experiences <span class="emphasis"><em>non-uniform</em></span>
      memory access time.<br> As a consequence the best performance is achieved
      by keeping the memory access local.
    </p>
<p>
      <span class="inlinemediaobject"><img src="../../../../../libs/fiber/doc/NUMA.png" align="middle" alt="NUMA"></span>
    </p>
<h4>
<a name="fiber.numa.h0"></a>
      <span><a name="fiber.numa.numa_support_in_boost_fiber"></a></span><a class="link" href="numa.html#fiber.numa.numa_support_in_boost_fiber">NUMA
      support in Boost.Fiber</a>
    </h4>
<p>
      Because only a subset of the NUMA-functionality is exposed by several operating
      systems, Boost.Fiber provides only a minimalistic NUMA API.
    </p>
<div class="important"><table border="0" summary="Important">
<tr>
<td rowspan="2" align="center" valign="top" width="25"><img alt="[Important]" src="../../../../../doc/src/images/important.png"></td>
<th align="left">Important</th>
</tr>
<tr><td align="left" valign="top"><p>
        In order to enable NUMA support, b2 property <code class="computeroutput"><span class="identifier">numa</span><span class="special">=</span><span class="identifier">on</span></code> must
        be specified and linked against additional library <code class="computeroutput"><span class="identifier">libboost_fiber_numa</span><span class="special">.</span><span class="identifier">so</span></code>.
      </p></td></tr>
</table></div>
<div class="important"><table border="0" summary="Important">
<tr>
<td rowspan="2" align="center" valign="top" width="25"><img alt="[Important]" src="../../../../../doc/src/images/important.png"></td>
<th align="left">Important</th>
</tr>
<tr><td align="left" valign="top"><p>
        MinGW using pthread implementation is not supported on Windows.
      </p></td></tr>
</table></div>
<div class="table">
<a name="fiber.numa.supported_functionality_operating_systems"></a><p class="title"><b>Table&#160;1.1.&#160;Supported functionality/operating systems</b></p>
<div class="table-contents"><table class="table" summary="Supported functionality/operating systems">
<colgroup>
<col>
<col>
<col>
<col>
<col>
<col>
<col>
</colgroup>
<thead><tr>
<th>
            </th>
<th>
              <p>
                AIX
              </p>
            </th>
<th>
              <p>
                FreeBSD
              </p>
            </th>
<th>
              <p>
                HP/UX
              </p>
            </th>
<th>
              <p>
                Linux
              </p>
            </th>
<th>
              <p>
                Solaris
              </p>
            </th>
<th>
              <p>
                Windows
              </p>
            </th>
</tr></thead>
<tbody>
<tr>
<td>
              <p>
                pin thread
              </p>
            </td>
<td>
              <p>
                +
              </p>
            </td>
<td>
              <p>
                +
              </p>
            </td>
<td>
              <p>
                +
              </p>
            </td>
<td>
              <p>
                +
              </p>
            </td>
<td>
              <p>
                +
              </p>
            </td>
<td>
              <p>
                +
              </p>
            </td>
</tr>
<tr>
<td>
              <p>
                logical CPUs/NUMA nodes
              </p>
            </td>
<td>
              <p>
                +
              </p>
            </td>
<td>
              <p>
                +
              </p>
            </td>
<td>
              <p>
                +
              </p>
            </td>
<td>
              <p>
                +
              </p>
            </td>
<td>
              <p>
                +
              </p>
            </td>
<td>
              <p>
                +<sup>[<a name="fiber.numa.f1" href="#ftn.fiber.numa.f1" class="footnote">a</a>]</sup>
              </p>
            </td>
</tr>
<tr>
<td>
              <p>
                NUMA node distance
              </p>
            </td>
<td>
              <p>
                -
              </p>
            </td>
<td>
              <p>
                -
              </p>
            </td>
<td>
              <p>
                -
              </p>
            </td>
<td>
              <p>
                +
              </p>
            </td>
<td>
              <p>
                -
              </p>
            </td>
<td>
              <p>
                -
              </p>
            </td>
</tr>
<tr>
<td>
              <p>
                tested on
              </p>
            </td>
<td>
              <p>
                AIX 7.2
              </p>
            </td>
<td>
              <p>
                FreeBSD 11
              </p>
            </td>
<td>
              <p>
                -
              </p>
            </td>
<td>
              <p>
                Arch Linux (4.10.13)
              </p>
            </td>
<td>
              <p>
                OpenIndiana HIPSTER
              </p>
            </td>
<td>
              <p>
                Windows 10
              </p>
            </td>
</tr>
</tbody>
<tbody class="footnotes"><tr><td colspan="7"><div class="footnote"><p><sup>[<a name="ftn.fiber.numa.f1" href="#fiber.numa.f1" class="para">a</a>] </sup>
                  Windows organizes logical cpus in groups of 64; boost.fiber maps
                  {group-id,cpud-id} to a scalar equivalent to cpu ID of Linux (64
                  * group ID + cpu ID).
                </p></div></td></tr></tbody>
</table></div>
</div>
<br class="table-break"><p>
      In order to keep the memory access local as possible, the NUMA topology must
      be evaluated.
    </p>
<pre class="programlisting"><span class="identifier">std</span><span class="special">::</span><span class="identifier">vector</span><span class="special">&lt;</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">fibers</span><span class="special">::</span><span class="identifier">numa</span><span class="special">::</span><span class="identifier">node</span> <span class="special">&gt;</span> <span class="identifier">topo</span> <span class="special">=</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">fibers</span><span class="special">::</span><span class="identifier">numa</span><span class="special">::</span><span class="identifier">topology</span><span class="special">();</span>
<span class="keyword">for</span> <span class="special">(</span> <span class="keyword">auto</span> <span class="identifier">n</span> <span class="special">:</span> <span class="identifier">topo</span><span class="special">)</span> <span class="special">{</span>
    <span class="identifier">std</span><span class="special">::</span><span class="identifier">cout</span> <span class="special">&lt;&lt;</span> <span class="string">"node: "</span> <span class="special">&lt;&lt;</span> <span class="identifier">n</span><span class="special">.</span><span class="identifier">id</span> <span class="special">&lt;&lt;</span> <span class="string">" | "</span><span class="special">;</span>
    <span class="identifier">std</span><span class="special">::</span><span class="identifier">cout</span> <span class="special">&lt;&lt;</span> <span class="string">"cpus: "</span><span class="special">;</span>
    <span class="keyword">for</span> <span class="special">(</span> <span class="keyword">auto</span> <span class="identifier">cpu_id</span> <span class="special">:</span> <span class="identifier">n</span><span class="special">.</span><span class="identifier">logical_cpus</span><span class="special">)</span> <span class="special">{</span>
        <span class="identifier">std</span><span class="special">::</span><span class="identifier">cout</span> <span class="special">&lt;&lt;</span> <span class="identifier">cpu_id</span> <span class="special">&lt;&lt;</span> <span class="string">" "</span><span class="special">;</span>
    <span class="special">}</span>
    <span class="identifier">std</span><span class="special">::</span><span class="identifier">cout</span> <span class="special">&lt;&lt;</span> <span class="string">"| distance: "</span><span class="special">;</span>
    <span class="keyword">for</span> <span class="special">(</span> <span class="keyword">auto</span> <span class="identifier">d</span> <span class="special">:</span> <span class="identifier">n</span><span class="special">.</span><span class="identifier">distance</span><span class="special">)</span> <span class="special">{</span>
        <span class="identifier">std</span><span class="special">::</span><span class="identifier">cout</span> <span class="special">&lt;&lt;</span> <span class="identifier">d</span> <span class="special">&lt;&lt;</span> <span class="string">" "</span><span class="special">;</span>
    <span class="special">}</span>
    <span class="identifier">std</span><span class="special">::</span><span class="identifier">cout</span> <span class="special">&lt;&lt;</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">endl</span><span class="special">;</span>
<span class="special">}</span>
<span class="identifier">std</span><span class="special">::</span><span class="identifier">cout</span> <span class="special">&lt;&lt;</span> <span class="string">"done"</span> <span class="special">&lt;&lt;</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">endl</span><span class="special">;</span>

<span class="identifier">output</span><span class="special">:</span>
    <span class="identifier">node</span><span class="special">:</span> <span class="number">0</span> <span class="special">|</span> <span class="identifier">cpus</span><span class="special">:</span> <span class="number">0</span> <span class="number">1</span> <span class="number">2</span> <span class="number">3</span> <span class="number">4</span> <span class="number">5</span> <span class="number">6</span> <span class="number">7</span> <span class="number">16</span> <span class="number">17</span> <span class="number">18</span> <span class="number">19</span> <span class="number">20</span> <span class="number">21</span> <span class="number">22</span> <span class="number">23</span> <span class="special">|</span> <span class="identifier">distance</span><span class="special">:</span> <span class="number">10</span> <span class="number">21</span>
    <span class="identifier">node</span><span class="special">:</span> <span class="number">1</span> <span class="special">|</span> <span class="identifier">cpus</span><span class="special">:</span> <span class="number">8</span> <span class="number">9</span> <span class="number">10</span> <span class="number">11</span> <span class="number">12</span> <span class="number">13</span> <span class="number">14</span> <span class="number">15</span> <span class="number">24</span> <span class="number">25</span> <span class="number">26</span> <span class="number">27</span> <span class="number">28</span> <span class="number">29</span> <span class="number">30</span> <span class="number">31</span> <span class="special">|</span> <span class="identifier">distance</span><span class="special">:</span> <span class="number">21</span> <span class="number">10</span>
    <span class="identifier">done</span>
</pre>
<p>
      The example shows that the systems consits out of 2 NUMA-nodes, to each NUMA-node
      belong 16 logical cpus. The distance measures the costs to access the memory
      of another NUMA-node. A NUMA-node has always a distance <code class="computeroutput"><span class="number">10</span></code>
      to itself (lowest possible value).<br> The position in the array corresponds
      with the NUMA-node ID.
    </p>
<p>
      Some work-loads benefit from pinning threads to a logical cpus. For instance
      scheduling algorithm <a class="link" href="numa.html#class_numa_work_stealing"><code class="computeroutput">numa::work_stealing</code></a> pins the thread
      that runs the fiber scheduler to a logical cpu. This prevents the operating
      system scheduler to move the thread to another logical cpu that might run other
      fiber scheduler(s) or migrating the thread to a logical cpu part of another
      NUMA-node.
    </p>
<pre class="programlisting"><span class="keyword">void</span> <span class="identifier">thread</span><span class="special">(</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span> <span class="identifier">cpu_id</span><span class="special">,</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span> <span class="identifier">node_id</span><span class="special">,</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">vector</span><span class="special">&lt;</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">fibers</span><span class="special">::</span><span class="identifier">numa</span><span class="special">::</span><span class="identifier">node</span> <span class="special">&gt;</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">topo</span><span class="special">)</span> <span class="special">{</span>
    <span class="comment">// thread registers itself at work-stealing scheduler</span>
    <span class="identifier">boost</span><span class="special">::</span><span class="identifier">fibers</span><span class="special">::</span><span class="identifier">use_scheduling_algorithm</span><span class="special">&lt;</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">fibers</span><span class="special">::</span><span class="identifier">algo</span><span class="special">::</span><span class="identifier">numa</span><span class="special">::</span><span class="identifier">work_stealing</span> <span class="special">&gt;(</span> <span class="identifier">cpu_id</span><span class="special">,</span> <span class="identifier">node_id</span><span class="special">,</span> <span class="identifier">topo</span><span class="special">);</span>
    <span class="special">...</span>
<span class="special">}</span>

<span class="comment">// evaluate the NUMA topology</span>
<span class="identifier">std</span><span class="special">::</span><span class="identifier">vector</span><span class="special">&lt;</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">fibers</span><span class="special">::</span><span class="identifier">numa</span><span class="special">::</span><span class="identifier">node</span> <span class="special">&gt;</span> <span class="identifier">topo</span> <span class="special">=</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">fibers</span><span class="special">::</span><span class="identifier">numa</span><span class="special">::</span><span class="identifier">topology</span><span class="special">();</span>
<span class="comment">// start-thread runs on NUMA-node `0`</span>
<span class="keyword">auto</span> <span class="identifier">node</span> <span class="special">=</span> <span class="identifier">topo</span><span class="special">[</span><span class="number">0</span><span class="special">];</span>
<span class="comment">// start-thread is pinnded to first cpu ID in the list of logical cpus of NUMA-node `0`</span>
<span class="keyword">auto</span> <span class="identifier">start_cpu_id</span> <span class="special">=</span> <span class="special">*</span> <span class="identifier">node</span><span class="special">.</span><span class="identifier">logical_cpus</span><span class="special">.</span><span class="identifier">begin</span><span class="special">();</span>
<span class="comment">// start worker-threads first</span>
<span class="identifier">std</span><span class="special">::</span><span class="identifier">vector</span><span class="special">&lt;</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">thread</span> <span class="special">&gt;</span> <span class="identifier">threads</span><span class="special">;</span>
<span class="keyword">for</span> <span class="special">(</span> <span class="keyword">auto</span> <span class="special">&amp;</span> <span class="identifier">node</span> <span class="special">:</span> <span class="identifier">topo</span><span class="special">)</span> <span class="special">{</span>
    <span class="keyword">for</span> <span class="special">(</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span> <span class="identifier">cpu_id</span> <span class="special">:</span> <span class="identifier">node</span><span class="special">.</span><span class="identifier">logical_cpus</span><span class="special">)</span> <span class="special">{</span>
        <span class="comment">// exclude start-thread</span>
        <span class="keyword">if</span> <span class="special">(</span> <span class="identifier">start_cpu_id</span> <span class="special">!=</span> <span class="identifier">cpu_id</span><span class="special">)</span> <span class="special">{</span>
            <span class="comment">// spawn thread</span>
            <span class="identifier">threads</span><span class="special">.</span><span class="identifier">emplace_back</span><span class="special">(</span> <span class="identifier">thread</span><span class="special">,</span> <span class="identifier">cpu_id</span><span class="special">,</span> <span class="identifier">node</span><span class="special">.</span><span class="identifier">id</span><span class="special">,</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">cref</span><span class="special">(</span> <span class="identifier">topo</span><span class="special">)</span> <span class="special">);</span>
        <span class="special">}</span>
    <span class="special">}</span>
<span class="special">}</span>
<span class="comment">// start-thread registers itself on work-stealing scheduler</span>
<span class="identifier">boost</span><span class="special">::</span><span class="identifier">fibers</span><span class="special">::</span><span class="identifier">use_scheduling_algorithm</span><span class="special">&lt;</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">fibers</span><span class="special">::</span><span class="identifier">algo</span><span class="special">::</span><span class="identifier">numa</span><span class="special">::</span><span class="identifier">work_stealing</span> <span class="special">&gt;(</span> <span class="identifier">start_cpu_id</span><span class="special">,</span> <span class="identifier">node</span><span class="special">.</span><span class="identifier">id</span><span class="special">,</span> <span class="identifier">topo</span><span class="special">);</span>
<span class="special">...</span>
</pre>
<p>
      The example evaluates the NUMA topology with <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">fibers</span><span class="special">::</span><span class="identifier">numa</span><span class="special">::</span><span class="identifier">topology</span><span class="special">()</span></code>
      and spawns for each logical cpu a thread. Each spawned thread installs the
      NUMA-aware work-stealing scheduler. The scheduler pins the thread to the logical
      cpu that was specified at construction.<br> If the local queue of one thread
      runs out of ready fibers, the thread tries to steal a ready fiber from another
      thread running at logical cpu that belong to the same NUMA-node (local memory
      access). If no fiber could be stolen, the thread tries to steal fibers from
      logical cpus part of other NUMA-nodes (remote memory access).
    </p>
<h4>
<a name="fiber.numa.h1"></a>
      <span><a name="fiber.numa.synopsis"></a></span><a class="link" href="numa.html#fiber.numa.synopsis">Synopsis</a>
    </h4>
<pre class="programlisting"><span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">fiber</span><span class="special">/</span><span class="identifier">numa</span><span class="special">/</span><span class="identifier">pin_thread</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
<span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">fiber</span><span class="special">/</span><span class="identifier">numa</span><span class="special">/</span><span class="identifier">topology</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>

<span class="keyword">namespace</span> <span class="identifier">boost</span> <span class="special">{</span>
<span class="keyword">namespace</span> <span class="identifier">fibers</span> <span class="special">{</span>
<span class="keyword">namespace</span> <span class="identifier">numa</span> <span class="special">{</span>

<span class="keyword">struct</span> <span class="identifier">node</span> <span class="special">{</span>
    <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span>                   <span class="identifier">id</span><span class="special">;</span>
    <span class="identifier">std</span><span class="special">::</span><span class="identifier">set</span><span class="special">&lt;</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span> <span class="special">&gt;</span>       <span class="identifier">logical_cpus</span><span class="special">;</span>
    <span class="identifier">std</span><span class="special">::</span><span class="identifier">vector</span><span class="special">&lt;</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span> <span class="special">&gt;</span>    <span class="identifier">distance</span><span class="special">;</span>
<span class="special">};</span>
<span class="keyword">bool</span> <span class="keyword">operator</span><span class="special">&lt;(</span> <span class="identifier">node</span> <span class="keyword">const</span><span class="special">&amp;,</span> <span class="identifier">node</span> <span class="keyword">const</span><span class="special">&amp;)</span> <span class="keyword">noexcept</span><span class="special">;</span>

<span class="identifier">std</span><span class="special">::</span><span class="identifier">vector</span><span class="special">&lt;</span> <span class="identifier">node</span> <span class="special">&gt;</span> <span class="identifier">topology</span><span class="special">();</span>

<span class="keyword">void</span> <span class="identifier">pin_thread</span><span class="special">(</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span><span class="special">);</span>
<span class="keyword">void</span> <span class="identifier">pin_thread</span><span class="special">(</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span><span class="special">,</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">thread</span><span class="special">::</span><span class="identifier">native_handle_type</span><span class="special">);</span>

<span class="special">}}}</span>

<span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">fiber</span><span class="special">/</span><span class="identifier">numa</span><span class="special">/</span><span class="identifier">algo</span><span class="special">/</span><span class="identifier">work_stealing</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>

<span class="keyword">namespace</span> <span class="identifier">boost</span> <span class="special">{</span>
<span class="keyword">namespace</span> <span class="identifier">fibers</span> <span class="special">{</span>
<span class="keyword">namespace</span> <span class="identifier">numa</span> <span class="special">{</span>
<span class="keyword">namespace</span> <span class="identifier">algo</span> <span class="special">{</span>

<span class="keyword">class</span> <span class="identifier">work_stealing</span><span class="special">;</span>

<span class="special">}}}</span>
</pre>
<p>
      </p>
<h5>
<a name="class_numa_node_bridgehead"></a>
  <span><a name="class_numa_node"></a></span>
  <a class="link" href="numa.html#class_numa_node">Class <code class="computeroutput">numa::node</code></a>
</h5>
<p>
    </p>
<pre class="programlisting"><span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">fiber</span><span class="special">/</span><span class="identifier">numa</span><span class="special">/</span><span class="identifier">topology</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>

<span class="keyword">namespace</span> <span class="identifier">boost</span> <span class="special">{</span>
<span class="keyword">namespace</span> <span class="identifier">fibers</span> <span class="special">{</span>
<span class="keyword">namespace</span> <span class="identifier">numa</span> <span class="special">{</span>

<span class="keyword">struct</span> <span class="identifier">node</span> <span class="special">{</span>
    <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span>                   <span class="identifier">id</span><span class="special">;</span>
    <span class="identifier">std</span><span class="special">::</span><span class="identifier">set</span><span class="special">&lt;</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span> <span class="special">&gt;</span>       <span class="identifier">logical_cpus</span><span class="special">;</span>
    <span class="identifier">std</span><span class="special">::</span><span class="identifier">vector</span><span class="special">&lt;</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span> <span class="special">&gt;</span>    <span class="identifier">distance</span><span class="special">;</span>
<span class="special">};</span>
<span class="keyword">bool</span> <span class="keyword">operator</span><span class="special">&lt;(</span> <span class="identifier">node</span> <span class="keyword">const</span><span class="special">&amp;,</span> <span class="identifier">node</span> <span class="keyword">const</span><span class="special">&amp;)</span> <span class="keyword">noexcept</span><span class="special">;</span>

<span class="special">}}}</span>
</pre>
<p>
      </p>
<h5>
<a name="numa_node_id_bridgehead"></a>
  <span><a name="numa_node_id"></a></span>
  <a class="link" href="numa.html#numa_node_id">Data member <code class="computeroutput">id</code></a>
</h5>
<p>
    </p>
<pre class="programlisting"><span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span> <span class="identifier">id</span><span class="special">;</span>
</pre>
<div class="variablelist">
<p class="title"><b></b></p>
<dl>
<dt><span class="term">Effects:</span></dt>
<dd><p>
            ID of the NUMA-node
          </p></dd>
</dl>
</div>
<p>
      </p>
<h5>
<a name="numa_node_logical_cpus_bridgehead"></a>
  <span><a name="numa_node_logical_cpus"></a></span>
  <a class="link" href="numa.html#numa_node_logical_cpus">Data
      member <code class="computeroutput">logical_cpus</code></a>
</h5>
<p>
    </p>
<pre class="programlisting"><span class="identifier">std</span><span class="special">::</span><span class="identifier">set</span><span class="special">&lt;</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span> <span class="special">&gt;</span> <span class="identifier">logical_cpus</span><span class="special">;</span>
</pre>
<div class="variablelist">
<p class="title"><b></b></p>
<dl>
<dt><span class="term">Effects:</span></dt>
<dd><p>
            set of logical cpu IDs belonging to the NUMA-node
          </p></dd>
</dl>
</div>
<p>
      </p>
<h5>
<a name="numa_node_distance_bridgehead"></a>
  <span><a name="numa_node_distance"></a></span>
  <a class="link" href="numa.html#numa_node_distance">Data member
      <code class="computeroutput">distance</code></a>
</h5>
<p>
    </p>
<pre class="programlisting"><span class="identifier">std</span><span class="special">::</span><span class="identifier">vector</span><span class="special">&lt;</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span> <span class="special">&gt;</span> <span class="identifier">distance</span><span class="special">;</span>
</pre>
<div class="variablelist">
<p class="title"><b></b></p>
<dl>
<dt><span class="term">Effects:</span></dt>
<dd><p>
            The distance between NUMA-nodes describe the cots of accessing the remote
            memory.
          </p></dd>
<dt><span class="term">Note:</span></dt>
<dd><p>
            A NUMA-node has a distance of <code class="computeroutput"><span class="number">10</span></code>
            to itself, remote NUMA-nodes have a distance &gt; <code class="computeroutput"><span class="number">10</span></code>.
            The index in the array corresponds to the ID <code class="computeroutput"><span class="identifier">id</span></code>
            of the NUMA-node. At the moment only Linux returns the correct distances,
            for all other operating systems remote NUMA-nodes get a default value
            of <code class="computeroutput"><span class="number">20</span></code>.
          </p></dd>
</dl>
</div>
<p>
      </p>
<h5>
<a name="numa_node_operator_less_bridgehead"></a>
  <span><a name="numa_node_operator_less"></a></span>
  <a class="link" href="numa.html#numa_node_operator_less">Member
      function <code class="computeroutput">operator&lt;</code>()</a>
</h5>
<p>
    </p>
<pre class="programlisting"><span class="keyword">bool</span> <span class="keyword">operator</span><span class="special">&lt;(</span> <span class="identifier">node</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">lhs</span><span class="special">,</span> <span class="identifier">node</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">rhs</span><span class="special">)</span> <span class="keyword">const</span> <span class="keyword">noexcept</span><span class="special">;</span>
</pre>
<div class="variablelist">
<p class="title"><b></b></p>
<dl>
<dt><span class="term">Returns:</span></dt>
<dd><p>
            <code class="computeroutput"><span class="keyword">true</span></code> if <code class="computeroutput"><span class="identifier">lhs</span>
            <span class="special">!=</span> <span class="identifier">rhs</span></code>
            is true and the implementation-defined total order of <code class="computeroutput"><span class="identifier">node</span><span class="special">::</span><span class="identifier">id</span></code>
            values places <code class="computeroutput"><span class="identifier">lhs</span></code> before
            <code class="computeroutput"><span class="identifier">rhs</span></code>, false otherwise.
          </p></dd>
<dt><span class="term">Throws:</span></dt>
<dd><p>
            Nothing.
          </p></dd>
</dl>
</div>
<p>
      </p>
<h5>
<a name="numa_topology_bridgehead"></a>
  <span><a name="numa_topology"></a></span>
  <a class="link" href="numa.html#numa_topology">Non-member function <code class="computeroutput">numa::topology()</code></a>
</h5>
<p>
    </p>
<pre class="programlisting"><span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">fiber</span><span class="special">/</span><span class="identifier">numa</span><span class="special">/</span><span class="identifier">topology</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>

<span class="keyword">namespace</span> <span class="identifier">boost</span> <span class="special">{</span>
<span class="keyword">namespace</span> <span class="identifier">fibers</span> <span class="special">{</span>
<span class="keyword">namespace</span> <span class="identifier">numa</span> <span class="special">{</span>

<span class="identifier">std</span><span class="special">::</span><span class="identifier">vector</span><span class="special">&lt;</span> <span class="identifier">node</span> <span class="special">&gt;</span> <span class="identifier">topology</span><span class="special">();</span>

<span class="special">}}}</span>
</pre>
<div class="variablelist">
<p class="title"><b></b></p>
<dl>
<dt><span class="term">Effects:</span></dt>
<dd><p>
            Evaluates the NUMA topology.
          </p></dd>
<dt><span class="term">Returns:</span></dt>
<dd><p>
            a vector of NUMA-nodes describing the NUMA architecture of the system
            (each element represents a NUMA-node).
          </p></dd>
<dt><span class="term">Throws:</span></dt>
<dd><p>
            <code class="computeroutput"><span class="identifier">system_error</span></code>
          </p></dd>
</dl>
</div>
<p>
      </p>
<h5>
<a name="numa_pin_thread_bridgehead"></a>
  <span><a name="numa_pin_thread"></a></span>
  <a class="link" href="numa.html#numa_pin_thread">Non-member function
      <code class="computeroutput">numa::pin_thread()</code></a>
</h5>
<p>
    </p>
<pre class="programlisting"><span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">fiber</span><span class="special">/</span><span class="identifier">numa</span><span class="special">/</span><span class="identifier">pin_thread</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>

<span class="keyword">namespace</span> <span class="identifier">boost</span> <span class="special">{</span>
<span class="keyword">namespace</span> <span class="identifier">fibers</span> <span class="special">{</span>
<span class="keyword">namespace</span> <span class="identifier">numa</span> <span class="special">{</span>

<span class="keyword">void</span> <span class="identifier">pin_thread</span><span class="special">(</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span> <span class="identifier">cpu_id</span><span class="special">);</span>
<span class="keyword">void</span> <span class="identifier">pin_thread</span><span class="special">(</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span> <span class="identifier">cpu_id</span><span class="special">,</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">thread</span><span class="special">::</span><span class="identifier">native_handle_type</span> <span class="identifier">h</span><span class="special">);</span>

<span class="special">}}}</span>
</pre>
<div class="variablelist">
<p class="title"><b></b></p>
<dl>
<dt><span class="term">Effects:</span></dt>
<dd><p>
            First version pins <code class="computeroutput"><span class="keyword">this</span> <span class="identifier">thread</span></code> to the logical cpu with ID
            <code class="computeroutput"><span class="identifier">cpu_id</span></code>, e.g. the operating
            system scheduler will not migrate the thread to another logical cpu.
            The second variant pins the thread with the native ID <code class="computeroutput"><span class="identifier">h</span></code>
            to logical cpu with ID <code class="computeroutput"><span class="identifier">cpu_id</span></code>.
          </p></dd>
<dt><span class="term">Throws:</span></dt>
<dd><p>
            <code class="computeroutput"><span class="identifier">system_error</span></code>
          </p></dd>
</dl>
</div>
<p>
      </p>
<h5>
<a name="class_numa_work_stealing_bridgehead"></a>
  <span><a name="class_numa_work_stealing"></a></span>
  <a class="link" href="numa.html#class_numa_work_stealing">Class
      <code class="computeroutput">numa::work_stealing</code></a>
</h5>
<p>
    </p>
<p>
      This class implements <a class="link" href="scheduling.html#class_algorithm"><code class="computeroutput">algorithm</code></a>; the thread running this scheduler
      is pinned to the given logical cpu. If the local ready-queue runs out of ready
      fibers, ready fibers are stolen from other schedulers that run on logical cpus
      that belong to the same NUMA-node (local memory access).<br> If no ready
      fibers can be stolen from the local NUMA-node, the algorithm selects schedulers
      running on other NUMA-nodes (remote memory access).<br> The victim scheduler
      (from which a ready fiber is stolen) is selected at random.
    </p>
<pre class="programlisting"><span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">fiber</span><span class="special">/</span><span class="identifier">numa</span><span class="special">/</span><span class="identifier">algo</span><span class="special">/</span><span class="identifier">work_stealing</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>

<span class="keyword">namespace</span> <span class="identifier">boost</span> <span class="special">{</span>
<span class="keyword">namespace</span> <span class="identifier">fibers</span> <span class="special">{</span>
<span class="keyword">namespace</span> <span class="identifier">numa</span> <span class="special">{</span>
<span class="keyword">namespace</span> <span class="identifier">algo</span> <span class="special">{</span>

<span class="keyword">class</span> <span class="identifier">work_stealing</span> <span class="special">:</span> <span class="keyword">public</span> <span class="identifier">algorithm</span> <span class="special">{</span>
<span class="keyword">public</span><span class="special">:</span>
    <span class="identifier">work_stealing</span><span class="special">(</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span> <span class="identifier">cpu_id</span><span class="special">,</span>
                   <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span> <span class="identifier">node_id</span><span class="special">,</span>
                   <span class="identifier">std</span><span class="special">::</span><span class="identifier">vector</span><span class="special">&lt;</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">fibers</span><span class="special">::</span><span class="identifier">numa</span><span class="special">::</span><span class="identifier">node</span> <span class="special">&gt;</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">topo</span><span class="special">,</span>
                   <span class="keyword">bool</span> <span class="identifier">suspend</span> <span class="special">=</span> <span class="keyword">false</span><span class="special">);</span>

    <span class="identifier">work_stealing</span><span class="special">(</span> <span class="identifier">work_stealing</span> <span class="keyword">const</span><span class="special">&amp;)</span> <span class="special">=</span> <span class="keyword">delete</span><span class="special">;</span>
    <span class="identifier">work_stealing</span><span class="special">(</span> <span class="identifier">work_stealing</span> <span class="special">&amp;&amp;)</span> <span class="special">=</span> <span class="keyword">delete</span><span class="special">;</span>

    <span class="identifier">work_stealing</span> <span class="special">&amp;</span> <span class="keyword">operator</span><span class="special">=(</span> <span class="identifier">work_stealing</span> <span class="keyword">const</span><span class="special">&amp;)</span> <span class="special">=</span> <span class="keyword">delete</span><span class="special">;</span>
    <span class="identifier">work_stealing</span> <span class="special">&amp;</span> <span class="keyword">operator</span><span class="special">=(</span> <span class="identifier">work_stealing</span> <span class="special">&amp;&amp;)</span> <span class="special">=</span> <span class="keyword">delete</span><span class="special">;</span>

    <span class="keyword">virtual</span> <span class="keyword">void</span> <span class="identifier">awakened</span><span class="special">(</span> <span class="identifier">context</span> <span class="special">*)</span> <span class="keyword">noexcept</span><span class="special">;</span>

    <span class="keyword">virtual</span> <span class="identifier">context</span> <span class="special">*</span> <span class="identifier">pick_next</span><span class="special">()</span> <span class="keyword">noexcept</span><span class="special">;</span>

    <span class="keyword">virtual</span> <span class="keyword">bool</span> <span class="identifier">has_ready_fibers</span><span class="special">()</span> <span class="keyword">const</span> <span class="keyword">noexcept</span><span class="special">;</span>

    <span class="keyword">virtual</span> <span class="keyword">void</span> <span class="identifier">suspend_until</span><span class="special">(</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">chrono</span><span class="special">::</span><span class="identifier">steady_clock</span><span class="special">::</span><span class="identifier">time_point</span> <span class="keyword">const</span><span class="special">&amp;)</span> <span class="keyword">noexcept</span><span class="special">;</span>

    <span class="keyword">virtual</span> <span class="keyword">void</span> <span class="identifier">notify</span><span class="special">()</span> <span class="keyword">noexcept</span><span class="special">;</span>
<span class="special">};</span>

<span class="special">}}}}</span>
</pre>
<h4>
<a name="fiber.numa.h2"></a>
      <span><a name="fiber.numa.constructor"></a></span><a class="link" href="numa.html#fiber.numa.constructor">Constructor</a>
    </h4>
<pre class="programlisting"><span class="identifier">work_stealing</span><span class="special">(</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span> <span class="identifier">cpu_id</span><span class="special">,</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">uint32_t</span> <span class="identifier">node_id</span><span class="special">,</span>
               <span class="identifier">std</span><span class="special">::</span><span class="identifier">vector</span><span class="special">&lt;</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">fibers</span><span class="special">::</span><span class="identifier">numa</span><span class="special">::</span><span class="identifier">node</span> <span class="special">&gt;</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">topo</span><span class="special">,</span>
               <span class="keyword">bool</span> <span class="identifier">suspend</span> <span class="special">=</span> <span class="keyword">false</span><span class="special">);</span>
</pre>
<div class="variablelist">
<p class="title"><b></b></p>
<dl>
<dt><span class="term">Effects:</span></dt>
<dd><p>
            Constructs work-stealing scheduling algorithm. The thread is pinned to
            logical cpu with ID <code class="computeroutput"><span class="identifier">cpu_id</span></code>.
            If local ready-queue runs out of ready fibers, ready fibers are stolen
            from other schedulers using <code class="computeroutput"><span class="identifier">topology</span></code>
            (represents the NUMA-topology of the system).
          </p></dd>
<dt><span class="term">Throws:</span></dt>
<dd><p>
            <code class="computeroutput"><span class="identifier">system_error</span></code>
          </p></dd>
<dt><span class="term">Note:</span></dt>
<dd><p>
            If <code class="computeroutput"><span class="identifier">suspend</span></code> is set to
            <code class="computeroutput"><span class="keyword">true</span></code>, then the scheduler
            suspends if no ready fiber could be stolen. The scheduler will by woken
            up if a sleeping fiber times out or it was notified from remote (other
            thread or fiber scheduler).
          </p></dd>
</dl>
</div>
<p>
      </p>
<h5>
<a name="numa_work_stealing_awakened_bridgehead"></a>
  <span><a name="numa_work_stealing_awakened"></a></span>
  <a class="link" href="numa.html#numa_work_stealing_awakened">Member
      function <code class="computeroutput">awakened</code>()</a>
</h5>
<p>
    </p>
<pre class="programlisting"><span class="keyword">virtual</span> <span class="keyword">void</span> <span class="identifier">awakened</span><span class="special">(</span> <span class="identifier">context</span> <span class="special">*</span> <span class="identifier">f</span><span class="special">)</span> <span class="keyword">noexcept</span><span class="special">;</span>
</pre>
<div class="variablelist">
<p class="title"><b></b></p>
<dl>
<dt><span class="term">Effects:</span></dt>
<dd><p>
            Enqueues fiber <code class="computeroutput"><span class="identifier">f</span></code> onto
            the shared ready queue.
          </p></dd>
<dt><span class="term">Throws:</span></dt>
<dd><p>
            Nothing.
          </p></dd>
</dl>
</div>
<p>
      </p>
<h5>
<a name="numa_work_stealing_pick_next_bridgehead"></a>
  <span><a name="numa_work_stealing_pick_next"></a></span>
  <a class="link" href="numa.html#numa_work_stealing_pick_next">Member
      function <code class="computeroutput">pick_next</code>()</a>
</h5>
<p>
    </p>
<pre class="programlisting"><span class="keyword">virtual</span> <span class="identifier">context</span> <span class="special">*</span> <span class="identifier">pick_next</span><span class="special">()</span> <span class="keyword">noexcept</span><span class="special">;</span>
</pre>
<div class="variablelist">
<p class="title"><b></b></p>
<dl>
<dt><span class="term">Returns:</span></dt>
<dd><p>
            the fiber at the head of the ready queue, or <code class="computeroutput"><span class="keyword">nullptr</span></code>
            if the queue is empty.
          </p></dd>
<dt><span class="term">Throws:</span></dt>
<dd><p>
            Nothing.
          </p></dd>
<dt><span class="term">Note:</span></dt>
<dd><p>
            Placing ready fibers onto the tail of the sahred queue, and returning
            them from the head of that queue, shares the thread between ready fibers
            in round-robin fashion.
          </p></dd>
</dl>
</div>
<p>
      </p>
<h5>
<a name="numa_work_stealing_has_ready_fibers_bridgehead"></a>
  <span><a name="numa_work_stealing_has_ready_fibers"></a></span>
  <a class="link" href="numa.html#numa_work_stealing_has_ready_fibers">Member
      function <code class="computeroutput">has_ready_fibers</code>()</a>
</h5>
<p>
    </p>
<pre class="programlisting"><span class="keyword">virtual</span> <span class="keyword">bool</span> <span class="identifier">has_ready_fibers</span><span class="special">()</span> <span class="keyword">const</span> <span class="keyword">noexcept</span><span class="special">;</span>
</pre>
<div class="variablelist">
<p class="title"><b></b></p>
<dl>
<dt><span class="term">Returns:</span></dt>
<dd><p>
            <code class="computeroutput"><span class="keyword">true</span></code> if scheduler has fibers
            ready to run.
          </p></dd>
<dt><span class="term">Throws:</span></dt>
<dd><p>
            Nothing.
          </p></dd>
</dl>
</div>
<p>
      </p>
<h5>
<a name="numa_work_stealing_suspend_until_bridgehead"></a>
  <span><a name="numa_work_stealing_suspend_until"></a></span>
  <a class="link" href="numa.html#numa_work_stealing_suspend_until">Member
      function <code class="computeroutput">suspend_until</code>()</a>
</h5>
<p>
    </p>
<pre class="programlisting"><span class="keyword">virtual</span> <span class="keyword">void</span> <span class="identifier">suspend_until</span><span class="special">(</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">chrono</span><span class="special">::</span><span class="identifier">steady_clock</span><span class="special">::</span><span class="identifier">time_point</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">abs_time</span><span class="special">)</span> <span class="keyword">noexcept</span><span class="special">;</span>
</pre>
<div class="variablelist">
<p class="title"><b></b></p>
<dl>
<dt><span class="term">Effects:</span></dt>
<dd><p>
            Informs <code class="computeroutput"><span class="identifier">work_stealing</span></code>
            that no ready fiber will be available until time-point <code class="computeroutput"><span class="identifier">abs_time</span></code>. This implementation blocks
            in <a href="http://en.cppreference.com/w/cpp/thread/condition_variable/wait_until" target="_top"><code class="computeroutput"><span class="identifier">std</span><span class="special">::</span><span class="identifier">condition_variable</span><span class="special">::</span><span class="identifier">wait_until</span><span class="special">()</span></code></a>.
          </p></dd>
<dt><span class="term">Throws:</span></dt>
<dd><p>
            Nothing.
          </p></dd>
</dl>
</div>
<p>
      </p>
<h5>
<a name="numa_work_stealing_notify_bridgehead"></a>
  <span><a name="numa_work_stealing_notify"></a></span>
  <a class="link" href="numa.html#numa_work_stealing_notify">Member
      function <code class="computeroutput">notify</code>()</a>
</h5>
<p>
    </p>
<pre class="programlisting"><span class="keyword">virtual</span> <span class="keyword">void</span> <span class="identifier">notify</span><span class="special">()</span> <span class="keyword">noexcept</span> <span class="special">=</span> <span class="number">0</span><span class="special">;</span>
</pre>
<div class="variablelist">
<p class="title"><b></b></p>
<dl>
<dt><span class="term">Effects:</span></dt>
<dd><p>
            Wake up a pending call to <a class="link" href="scheduling.html#work_stealing_suspend_until"><code class="computeroutput">work_stealing::suspend_until()</code></a>,
            some fibers might be ready. This implementation wakes <code class="computeroutput"><span class="identifier">suspend_until</span><span class="special">()</span></code> via <a href="http://en.cppreference.com/w/cpp/thread/condition_variable/notify_all" target="_top"><code class="computeroutput"><span class="identifier">std</span><span class="special">::</span><span class="identifier">condition_variable</span><span class="special">::</span><span class="identifier">notify_all</span><span class="special">()</span></code></a>.
          </p></dd>
<dt><span class="term">Throws:</span></dt>
<dd><p>
            Nothing.
          </p></dd>
</dl>
</div>
<div class="footnotes">
<br><hr width="100" align="left">
<div class="footnote"><p><sup>[<a name="ftn.fiber.numa.f0" href="#fiber.numa.f0" class="para">8</a>] </sup>
        On x86 the interconnection is implemented by Intel's Quick Path Interconnect
        (QPI) and AMD's HyperTransport.
      </p></div>
</div>
</div>
<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
<td align="left"></td>
<td align="right"><div class="copyright-footer">Copyright &#169; 2013 Oliver Kowalke<p>
        Distributed under the Boost Software License, Version 1.0. (See accompanying
        file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>)
      </p>
</div></td>
</tr></table>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="speculation.html"><img src="../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../index.html"><img src="../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../index.html"><img src="../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="gpu_computing.html"><img src="../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
</body>
</html>
